Effective Linear Discriminant Analysis for High Dimensional, Low Sample Size Data

In the so-called high dimensional, low sample size (HDLSS) settings, LDA possesses the “data piling” property, that is, it maps all points from the same class in the training data to a common point, and so when viewed along the LDA projection directions, the data are piled up. Data piling indicates overfitting and usually results in poor out-of-sample classification. In this paper, a novel approach to overcome the data piling problem is introduced. It incorporates variable selection into LDA. The underlying assumption is that, among the large number of variables there are many irrelevant or redundant variables for the purpose of classification. By using only important or significant variables we essentially deal with a lower dimensional problem. Experiments on both synthetic and real data sets show that the proposed method is effective in overcoming the data piling and overfitting problem of LDA while improving the out-of-sample classification performance.